The two foundations
Almost everything in Databricks data engineering rests on these. Learn the stages on top of them.
Delta Lake
The storage format that makes data reliable — like a spreadsheet that never corrupts, remembers its history, and supports updates & deletes.
Unity Catalog
The governance layer that controls who can see and do what — across every table, file, and AI asset, using catalog.schema.table.
Scope of this plan
This plan focuses on the Databricks data-engineering platform (Lakeflow + Delta + Unity Catalog). Two things are intentionally left out so you can spend your time on platform skills:
Workspace basics — skipped
Signing up, the UI tour, clusters/compute setup, notebooks & magic commands, Git Folders setup. Assumed as prerequisites.
Apache Spark core — skipped
PySpark/Spark SQL programming, DataFrame transformations & joins, UDFs, higher-order functions, and the Spark execution model. Learn these separately if needed.
The roadmap
Click a stage to expand it. Tick the box when you've finished. Ask your tutor for a deep-dive notebook or lesson on any stage.
A note on naming
In June 2025 Databricks grouped its data engineering tools under one umbrella called Lakeflow. Old names still appear in many docs — here's the translation:
Delta Live Tables (DLT) → Lakeflow Spark Declarative Pipelines (SDP)
Workflows → Lakeflow Jobs
(managed connectors) → Lakeflow Connect
(new) → Lakeflow Designer — visual, no-code / natural-language pipeline builder
Existing DLT pipelines keep working unchanged — it's mostly a rebrand plus new capabilities. Lakeflow (Connect · SDP · Designer · Jobs) is the umbrella for end-to-end data engineering.